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NTRODl  CTloN 

I I i Air ••iiiMf ir  1 >at a Rroce.-sing  l.quipment  Selection  Office 
VDI’KSO;  7 the  Department  of  tie  Navy  is  cngugcd  in  a 
1<  vc|< -pmoni  program  for  soft w an-  to  In*  used  in  tin*  quant  i 
icaftoii  of  (output,  r ystoms  selection  criteria,  and  the  :ip- 
>!;ratioi,  •:  quality  control  procedure*  to  selected  software 
>r«  kIucIs. 

That  o ti  a j r gran,  he  undertaken  by  this  centralized 
\1  I I S«  It •«  1 1< . < ) f ? i * • t is  both  proper  and  important  to  thi- 
ll mission  This  mission,  briefly 

tat<*d.  i.~  to  evaluate  and  select,  or  review  the  selection  of, 
iimncr  iallv  available  automatic  data  processing  equipment 
or  approval  by  th<  Assistant  Secretary  of  the  Navy  for 
i inanciai  Management 

'Phis  development  program  is  the  responsibility  of  AI) 
*KS<  )*s  Soft  ware  1 )eveloptm  tit  I )m-ion,  ami  is  concent  rated 
n thre*  areas  A COBOL  Compiler  Validation  System  has 
>een  designed  implemented,  and  is  being  used  throughout 
In  bederal  C«  v rnment  to  determine  the  degree  to  which 
Y)BOI  i umpilers  conform  to  tin  published  landau!  A 
system  to  facilitate  the  process  of  (’OBOJ  benchmark  pro- 
gram converse -ii,  evaluation,  and  implementation  has  been 
•ompleted  and  i-  being  field  tested.  Finally,  tin  experiment  in 
i-mg  a library  of  synthetic  program-  for  system  perfornmm  • 
neasurenient  i-  being  conducted. 

\n  evaluation  of  such  a program  requires  a description  of 
he  projects,  the  identification  of  project  controls  which  have 
Men  applied,  and  flic  resultant  or  expected  payoffs  These 
a ill  b»  discussed  in  turn 

Why  a 7 utility  rontrol  pn>yratnf 

The  problem  we  an*  attempting  to  alleviate  is  a financial 
me  During  fiscal  year  1973  ADRKSO  participated  in  189 
Acquisition  a<  tmns  with  a monthly  rental  value  of  $691  (HM) 
I’his  docs  not  include  173  rent ilizat ion  aeti  -ns  'Hie  scope  of 
these  action-  is  (juite  broad  Recent  acquisitions  have  in- 
cluded Hitt  mini-computer  configurations.  .r>0  key-to-disk 
configurations,  and  a medical  laboratory  information  system 

How  do  the  above  dollar  figures  relat#  to  software?  Pn* 
rise  measurements  nr<-  difficult,  but  we  estimate  that  the 
Department  of  tie  Navy’s  annual  software  expenditure  is 


approximately  three  times  that  of  hardware,  Bair.  Boehm 
has  cited  a similar  figure  for  the  I S Air  l-orc.*  -md 
suspect  this  hgure  is  fairly  universal 

Our  present  work  has  as  one  of  it'  principal  ; up*  * ! 
lowering  of  software  production  and  n.uin: ■••nat.* 

Thes«  costs  will  of  (nurse  vary  with  the  natiir-  . ■ n 

m question  \ 1964  SIX  ' repot : suggested  that  appr  .\in  .r 
19  man-niouths  wen*  required  f*,r  the  d -livery  • HXXt  t. 

( hino  languagi  inst  ructions  Hu  lata  wet  lei 
projects,  and  included  program  d«.-gn.  du  g.  o.  I testing 
time.  The  incremental  tine-  per  1000  addin-  r.al  Inns  --I  <|. 

was  a man-months  (’orbato’s  data  gath*  red  Irnn  t*-  Mu!;  i«  - 
project*  indicate  that  productivity  can  Is-  v t - * 1 -i.  ■ reused 

through  the  use  of  a high*  r 1-  vt  1 languag.  . but  - ratir.  - 1 1 1 
remains  an  expensive  product. 

Much  of  the  high  oftwa  cost  is  t he  i suit 

and  of  conversion  costs  William.-4  has  reported  n a a 
version  project  undertaken  in  1964  by  tin  Dx-klie-  I Missih-s 
and  Space  Company  Tic  220  FORTRAN  progra:  whe  li 

were  converg'd,  from  an  IBM  7094  to  i l XI  VAC  lifts, 
r>  quired  five  month  for  flu  j b,  at  a « ■ ■.- 1 of  .,p,  v it. 
$141,000.  do  alleviate*  tin-  problem  we  -«  d t - . on  tin 
degree  to  which  higher  |.  vl  language  translators  n r • • ; f » 
published  standards,  and  w«  could  certainty  u-  m.  re  in  die 
way  of  conversion  mkIs,  particularly  data  * inv*  - u 

Finally,  we  have  found  that  the  entire  c<  mpetiti.-  • 1- 
tion  time  can  be  disturbingly  long  nine  to  23  month-  m ovir 
experience.  We  say  “disturbingly  b<*caus«*  a long 
process  is  evpensiv"  for  both  buyer  and  vendor  W ir** 
interested  in  software  which  could  perhaps  lx-  used  m -hot  m 
mg  the  time  span. 

THK  PROJECTS 

The  goals 

'the  user  of  high*  r h vel  languages  in  software  dev*  1- -i •nu-i  t 
will  reduce  tin*  cf<t  of  such  development  principally  by  in- 
creasing programmer  productivity  Two  languages.  FOR- 
TRAN andCOBOl.  have  been  standardized  so  as  to  in- r - as< 
their  usefulness  Standardization  efforts  are  a Is.  under  w.v 
for  BASK'  and  IM.  1 If  standardization  is  indeed  to  Uar 
some  advantage*,  commercial  conipliers  must  adhere,  in  tin  ir 
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translation,  to  the  publishes!  **andard  The  adherence  to  a 
standard  must  include  Ian/  semantics  'where  unain- 
biguoiis  as  w*  II  a>  language  s\  n».»\  I fT* *«  f ; \ . implementation 
of  a standard  requires  a means  "f  me? i^ir  ’he  degree  to 

which  compiler*  conform  to  the-  >tan<lard  Hills,  tin*  de-velojv- 
nu*nt,  u >»•  and  maintenance  of  a validation  system  for  CO- 
HOI  compilers  has  b«*on  an  important  effort  on  tin*  part  of 
th*  S«ft ware  Development  Division 

portability  is  -t  measure-  of  the  ease  of  moving  a computer 
program  from  on*  environment  to  another  Many  factors 
affect  a program’s  |*»rtahilif\  the  computer  system,  the 
language  n*ed.  program  design  ..I  the  application  At  this 
tinn  "<  arc  sp(*cificallv  mt«  -rested  in  COBOL  program 
portability  \ ( '<  >B<  )L  program  would  be  completely  jairtable 
if  all  non  standard  functions  eg.  extensions  to  the  lan- 
guage)  could  be  reduced  to  standard  functions,  all  imple- 
ment oj  nan.es  could  be  resolved,  data  representation  were 
standardized  acr-  s>  compnt  r -ystems.  and  no  “implementor 
defined”  language  (dements  «»n'  used  Practically.  this 
means  that  a eornplefel\  portable  COBOL  program  is  a 
figment  of  the  imagination!  W.  can.  however,  greatlv  im- 
pn»ve  a program’s  portability  by  developing  software  which 
addresses  itself  to  the  above  problems 

It  i*  also  important  that  we  not  sacrifice  too  much  ef- 
ficiency for  »le  sake  of  |M»rt abilit > \ recent  study  by  Inter- 

national Computer  Systems.  Inc  h indicate*  that  COBOL 
programs  are  g*  ner.-illy  -isi*  r to  convert  fto  other  COBOL 
dialects,  than  programs  in  IORTILW  or  assembly  lan- 
guage'. l*nfortunatel\ , the  same  study  also  indicates  that  the 
relative  operating  costs  of  converted  COBOL  programs  are 
nuicfi  higher  than  those  for  other  languages.  Our  aim  is  to 
achieve  significant  portability  at  a modest  cost  in  efficiency 
Because  such  an  aim  is  quite  relevant  to  benchmark  pro- 
grams we  refer  to  the  conversion  system  we  are  developing 
as  the  Benchmark  Preparation  System  (BPS). 

A significant  factor  contributing  to  d lays  in  computer 
systems  acquisition  has  been  the  preparation  and  processing 
time  of  user  benchmarks  Some  way  of  measuring  minimal 
system  throughput  capability  is  required  For  selection  pur- 
jM.ses,  benchmarks  an*  the  accepted  measurement  tool  in  the 
Department  of  the  Navy  The  major  problems  with  natural 
benchmarks  i e . existing  application  programs)  have  been 
the  following 

(a)  Kaeh  time  an  agency  selects  a system  a new  set  of 
benchmarks  is  prepared  'Hus  is  waste  ful 
(b  i The  benchmarks  are  often  not  debugged,  iuid  usually 
biased  toward  a given  architecture 
The  latter  problem  will  be  partially  alle  viates!  bv  the 
BJ'S  In  order  to  reduci  production  duplication  and  ooats  we 
are  developing  a ’ reference  benchmark  program  library.” 
This  is  a se-t  e#f  task-oriented  synthetic  programs  which  e*an 
be  used  individually  or  in  a mix,  in  conjunction  with  e>r  in- 
stead of  natural  Is-mhmarks 

Syxtrm*  to  fulfill  yonl* 

The  f ’<  *B<  )L  ( «»inpi|er  Validation  System  consists  of  audit 
routine*,  their  related  data,  and  an  executive  routine  I VP- 


roufine  t which  prepares  the  audit  routines  for  compilation  * 
I\ae*h  audit  routine  is  a COBOL  program  which  includes 
many  t<»st.-,  ami  supporting  procedure's  indicating  the  results 
of  the  tests  Tho  audit  rnijtitics  « »l|eet iv»-ly  contain  t h#* 
features  of  Standard  CuB<  l e xcept  f.,r  th<*  Keport  Writer 
module)  The  executive-  routine  auto mates  the  creation  of  a 
file*  containing  tic  audit  re»utine*  wiih  implementor  names 
inserted  in  the  soure*-  od<  :n,d  the-  "P« -rating  system  control 
cards  required  for  compiling  and  executing  each  routine.  The? 
testing  of  a compiler  m a particular  haniwan-  operating  svs- 
te»m  e-nvironnie-nt  i>  .mplished  by  compiling  and  executing 
each  audit  routine  Tin  input  report  pr-dured  by  eaeh 
routine  indicates  whethe*r  the-  compiler  passed  or  failed 
individually  1 the • test-  in  the  murine  If  the*  compiler  re  jects 
some  language  element  I » x terminating  compilation  (giving 
fatal  diagnostic  m<  -sag.  > (,r  te  rminating  execution  al>- 

normally  . tb«*n  the-  te--t  containing  the  the*  compiler  was 

unable*  to  proce-ss  i*  delet'd,  and  the  a dir  mutine  compiled 
again  A test  is  deleted  by  inserting  N()’l  \.  at  tin-  be-gmmng 
« if  the-  test  paragraph,  thereby  changing  the  source  e-ode  in  the 
test  paragraph  to  comment  -tat'*ni( nts  The  output  reports 
of  tin-  audit  routines  n-titepe-  the-  raw  data  from  which  the* 
members  e.f  the-  Federal  COBOL  Compiler  Testing  Service 
fan  activity  »*f  th-- Software  Development  Divisiot  produce  a 
Validation  Summary  l{ep«>rt.  which  provide*  » e-onsolidatid 
summary  of  t h« • result-  obtained  from  the-  validation  « »f  a 
compiler. 

The  results  of  running  the  COBOL  Compiler  Validation 
System  do  not  suggest  the  degree  to  which  th«  compiler  is 
unable  (\  e-  . capable  of  data  processing  applications)  but  the 
degree  to  which  individual  language  elements  are*  usable. 
This  will  give-  an  indication  of  conversions  which  will  he* 
necessary  in  order  t-  utiliz.  t sourer  program  from  another 
system  sup|>orririg  the  s.in.  language  s|*rilie-atie*ns  stand- 
ard. Thus,  the  Validation  '■*■  stem  ti  *.!*  a ('( >B(  >L  Compiler’s 
adhe  rence  t*»  the-  stamlarel  language  syntax,  and.  where  un- 
ambiguous, language  semantics  The-  latter  .if  course-  is  a more 
diffie-ult  area  ls»eause  <.f  the  lack  of  appropriate  mechanisms 
fe>r  pns’ise  semantic  at mn*  The  Validation  System 

dew*s  riot  evaluate*  the  mmlcme-ntat ion  of  -i  compiler  <i  e*  . is  it 
a text-in-com  or  -e»rnpi|.-r  in-core.  «-t e nor  it*  quantitative 

performance  ehara<  te-ri<tics 

Addii ie until v ■.  th»*  .summary  of  a validation  includes  an  imli- 
ratieui  e*f  unspe  -ei  ti(*d  language  semantics  tie-,  where  latitude 
is  given  fe»r  vendor  implementation ).  and  ambiguous  language* 
semantics,  l-inallv.  tables  summarizing  the-  running  time  and 
memory  utilization  of  the  audit  routines,  and  a characteriza- 
tion of  compiler  hard  copy  output  and  diagnostic*  an*  in- 
cluded in  Validation  Summary  Hearts. 

The  lienchmark  preparation  system  jH-rforms  conversion 
m the  major  areas  atTe*cting  port .ahi lit v e»f  appliratietn 
(’HBoL  program-  tie »a*t andard  ('DBoL  functions,  imple- 
mentor names,  and  data  representation  \ 1'OBOL  so\jrce 
preigram  translator  \ WTUAN-t')  take**  native  machine 
(’UBOL  preegram*  ami  converts  them  to  machine  indepen- 
dent roBOL  WSI  \ 1 23-UMi8  language  sjaviticatioiisl. 
Tboae  functions  in  the-  native  machine  OiBOL  which  are 
imteiiMonH  U»lhe-  ANSI  language*  s|ss-ification>  and  therefore 
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f.i  I r,\  rt « •»  t :»r  flagg*  -1  l»y  tin*  translator  fmple- 
•pt  i»t* -r  naim-s  in  the  hem  hmark  programs  ir-*  replaced  with 
miHjii-  naiitcs  in  the  machine  i»i<i#*pfii<i«*»it  source  programs 
I nM: i-  - ir  r<  gmz*d  ami  r«  plac’d  by  tin-  V P-Routiiic 

w 1 . i ),•  ;•  j.-  n.-  if  implemented  on  l hr  target  machine 
Input  data  t:i-  ..»oriaf.*d  with  the  benchmark  programs 
an  trat:  atni  bv  .*  -»  re  s <.f  (’OBnl,  programs  Phesr  data 
t r i -I  -ion  pr-  ‘grams  ma  k»  use  of  data  • *nvcrsi*  *n  Mibr-'ut  me> 
inherent  m the  respective  < *<  > H«  > I Compilers  native  <*r 
target  l ...  hue  m translating  th«  machine  depemlent  data 
• machine  independent  format  and  vice  versa  Machine 
lit ; end.  nt  data  characteristics  may  include  arithmetic  sign 
w<.rd  boundary  alignment,  and  certain  internal  rcpre-«nta 
tioi  I'h<  CoBol.  data  translation  programs  are  created 
from  the  benchmark  program  tile  descriptions.  The  creation 
i-  perf*  rim  d by  program  g-  iteration  I il**  descrip’mns  in  the 
data  translation  programs  are  those  fur  the  native  machine 
til  machine  independent  tile,  and  ANSI  target  fib*  I he 
native  n .wh  n.  • d«-<nption  i>  used  to  read  the  nativ* 
it  . i < riine  data  tiles  and  build  machine  independent  data  tiles 
■Ml  data  in  the-,  will  la  in  display  or  character  mode  with  tin 
sign-  of  numeric  data  stored  ><  oaratcly  Kssentiallv.  machine. 
dep«  r.dent  data  an  translated  to  a string  <>f  characters  which 
may  then  b.  -ohje*  t to  straight  character  code  translations 
for  the  appropriate  machine 

I’pon  transfer  of  the  data  files  to  the  target  machine,  the 
re  vers*  operation  occurs  The  madum  independent  data  are 
read  according  t • the  file  descriptions,  and  written  using  the 
\\S1  target  file  descriptions  Hm  data  translation  programs 
also  provide  tin  capability  « »f  validating  the  data  tiles  eg 
numerically  described  fields  wfiich  do  riot  contain  numeric 
data  are  identited  This  can  I »«•  done  by  a separate  execution 
r m conjunction  with  creating  the  independent  or  target 
miu  him*  data  tiles  The  benchmark  package  (programs  and 
data  lii‘  which  i>  dist rib' i ted  is  itself  in  machine  independent 
*.  form  Programs  are  in  a source  program  library  (Population 

| Pi;*  I hi  Population  Pile  contains  the  benchmark  source 

pr*  grams,  data  translation  program*  and  thi  \ P Routine. 

■ Prior  to  U*nchmark  processing  the  VP-Routine  selects  the 

nm*  him  independent  O >B<  >L  programs  front  the  imputation, 
inserts  the  necessary  f * d,  implementor  names  and  cr«  .it * - 
j a job  stream  tile  for  input  into  the  computer  system  The 

H \ P- Routine  als< » provides  the  U(Mlating  capability  f«»r  the 

Population  I »le  A summary  of  all  changes  made  to  t hi • 
f Populate,  j file  and  ♦he  job  control  language*  generated  for 

the  run  stream  file  is  part  of  the  output  create!  bv  th*  VP- 
Routine  Phis  summary  i-  used  to  determine  the  change-*  a 
vendor  has  to  make  in  implementing  the  benchmark  on  his 
system 

The  Reference  H«  m hrnurk  Programs  library  has  l»ecn 
iwd  in  performing  an  experin  ent  to  determine  the  suitability 
“f  synthetic  programs  ir»  alleviating  the  problems  created  bv 
natural  beriehmark-  I ive  processing  tasks  were  selerted  as 
representing  in  vary  ing  eombinations,  a large  numis»r  of  ap 
« plication  bisk-  Theae  were  sequential  file  processing,  in- 

dex**! sequential  file  processing  relative  1 O pn»c fusing, 
sorting,  arid  computation  COB<>!  programs  were  written  to 
perform  each  >t  these  t t-ks.  with  each  program  controlled 


by  a set  of  compile-time  and  c\e«  ution-t line  parameter*.  1 b 
ability  t • vary  automatically  ecrt  iin  parameters  at  eompil* 
time  provides  us  with  the  flexibility  to  develop  a fairlv  reii 
mix  from  just  a f.  w bade  pr*  grams 

We  have  found  * iirough  our  testing  ui*h  tic-*  pr  .gr 

that  a small  number  of  simple,  task-oriented,  > nth*  •»  - 
modules  can  be  combined  in t i versatile  j<|>  mix  \ rej, 
tively  small  number  *f  paranu •»  r-  is  sufficient  ! ••nab!*  i 

singl*  program  t > n fh-ct  thi  * haracf eristics  of  .»  hr*  .1  - - 

of  application-  Vb«  . individual  moduli'*  hav*  pr  v-  n us*  ' .1 
in  exercising  is* dated  computer  system  n atures,  such  as  I < ) 
handling  l inallv  if  one  acc«  pt>  a * modest  workload  • hara- 
tcrization,  aimed  m**re  at  reflecting  extremities  and  crucial 
are.-Ls  ratfn-r  than  rompn  lmn-iveness.  it  i>  jmssiblc  an  I 
reasonable  i » const  met  a benchmark  from  a set  of  .-yntheti 
rn*  Killies. 


PKoJlXT  ( < >NT!B  H.S 
Why 

Bochin’  has  suggested  that  t h«*  phrase  'soft wan*  engii  • r- 
ing"  is  a eont radict i> *n  in  terms  be*  aus*  we  h iv  e n data  I >.« 
to  b«*  used  in  measuring,  in  some  wav.  what  w « produce  and 
how  well  we  prinlucc  it  Vet,  his  *w n studii's  indicate  that  yy 
d.'*  have  some  data  to  w«»rk  with  In  order  to  obtain  more, 
those  «'f  us  yyhose  business  it  is  to  develop  - .ft war*  must 
keep  records  «»f  our  effort-  and  thereby  control  them.  'Phis 
does  not  present  an  undue  hardship  in  our  case  since  the 
Department  of  the  Navy  strongly  em  ourages  that  we  b«*  ac 
countable  f-*r  what  we  do.  how  we  do  it  and  what  it  is  worth 


// i>tr 


Because  we  are  a small  organization,  our  controls  are 
modest  but.  we  think,  effective. 

Much  has  been  made  « I ntru*  tureni  and  modular  program- 
ming 7 These  concepts  are  gaining  arceptanc  * in  (e.veiniiu  nt 
ami  private  industry  Wful*  w«*  take  no  issue  with  thi  ir  m*  nt 
we  would  suggest  eautiiui  in  their  applicability  M*slularitv 
will  often  n-duce  aoni**  “f  a system's  complexity,  but  max 
introduce  additional  complexity,  particularly  m the  inter- 
module connect  ions  I he  naturi  of  the  CUMol,  Validation 
Syst*  m dictates  that  it  b»  highly  modular,  but  we  have  found 
that  much  of  its  complexity  is  due  to  its  modularity  We  have 
also  found  that  c 1U' It M-ss  programming  car*  be*  awkward 
and.  especially  in  ('OBOL.  < ostlv  W . realie*^  that  deviations 
from  the  concepts  an*  “allowed."  but  then  we  are  back  to 
what  hav**  for  years  been  ree*»gniz*sl  as  simply  go«**|  program 
tning  praetieea 

VN«-  do  follow  modular  programming  concept*  as  design 
aids  This  se<*ms  t*>  have  become  a verv  common  practice  \ 
re* ent  Hoskyns  surv*  * f**r  tli**  British  (»*>v*  rnment  sh<»wed 
that  ffH  p#*rrent  «*f  modular  programming  practitioner*  did  s<, 
in  the  design  stage  \ major  benefit  of  this  practice  has  been 
a lowering  of  maintenance  costs 
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Figure  1 Pr<  ject  history  chart  for  synthetic  lxnrhmsrki> 

We've  considered  tin-  "lead  programmer "•  idea  and  dis- 
carded it  as  inapplicable  to  our  environment.  We  are  blessed 
with  a surplus  of  "lead''  programmers,  and  our  projects, 
while  sometimes  large,  as  in  the  case,  of  the  Validation  Sys 
tern,  ire  not  massive 

We  keep  records  of  our  work  All  "initial  project  form"  is 
used  to  identify  the  project  requestor,  the  purpose  and  nature 
of  the  project,  time  reipiircmeiits.  resources  r<-< piir< •»! . and  e\- 
pecled  pay.  ff  . generally  cannot  atTortl  the  luxury  of  con- 
tinuing it  resources  required  exceed  payoff,  in  dollars  We 
then  prepare  a work  plan.  This  includes  a schedule,  check 
(mints,  milestones,  and  man|*twcr  ris|Uircmeiit  distribution 
Milisloues  an  ills' inguish. si  front  checkf). lints  in  that  the 
former  require  a concrete  action  or  document  to  Is.  taken  or 
produced,  while  the  latter  may  simply  consist  of  an  indication 
that  ' parameter  testing  is  complete",  f igure  1 shows  the 
work  (linn  for  our  synthetic  program  library  project  It 
is  important  that  we  indicate  the  distribution  of  manpower 
over  the  project  lifespan,  since  this  enables  us  to  coordinate 
manpowir  nsjiiircmcnt*  for  several  projects  Wc  review  the 
workplans  whenever  we  feel  it  is  necessary  (but  at  least  at 
the  checkpoints  and  milwtom-s)  If  we  fall  liehind  we  revise 
the  workplan  Thus  far  we've  successfully  resisted  the  temp- 
tation to  add  manpower  or  adopt  unreasonable  catch-up 
schixiules  when  we  fall  Is  hind  The  necessity  for  such  resis- 
tant! has  Is*  n well  documented  by  Brooks"1  and  others. 

We  maintained  a log  of  compiler  errors  (computation,  se- 
quence control,  input  -out  ptit , etc.)  Imt  wo  abandoned  this 
U-cause  wc  dal  not  tmd  it  overly  useful  ( Iver  a sam(tlc  period 
of  five  months  wc  prisluced  approximately  10, 000  limn  of 
COBOL  code,  and  4'J  compile-time  errors.  Approximately 
half  of  these  were  clerical"  errors  'bad  keypunching,  sloppy 


printing,  etc.)  Recognizing  the  smallness  of  the  sample, 
would  still  make  the  generalization  that  any  overly  cxton-iv 
effort  iii  bis  fing  up  a compiler's  syntax  diagnostics  capabilit  > 
may  lie  a waste  of  time. 

A test  log  i.  kept  for  all  projects  The  log  indicates  which 
program  or  moduli  is  being  tested,  aims  of  tin  test,  whether 
these  were  achieved,  and  resources  ustsl.  The  sani  v. 
month  sample  showed  that  we  aehievtd  our  aims  m net 
under  b(t  pereent  of  the  tests,  and  that  new  problem  r. 
discovered  in  some  .10  percent  ot  the  tests  Also,  ill.  a vei  Ige 
test  run  usisl  less  than  three  memory  minutes  of  I \l  \ AC 
1 1 OH  time.  Ail  our  work  is  done  in  a remote  job  entry  I o, ' • lit 
mode.  Vet , the  above  figures  seem  to  imply  that  our  testing 
habits  are  more  consistent  with  what  would  be  x|>ect.  I ir  -i 
mtcraetive  program  development  enviroiiment  Suekii.u 
and  others  have  suggestisl  that  on-line  (irognminneg  u 
proves  ellielencv  It  appeal's  that,  additions  111  expel  i.i  . • d 
lirogrammers  tend  to  beliavi  as  if  they  were  m an  in.  ne 
environment  even  if  tic  i are  not 

Boehm’s’  statistics  indicate  that  4 . ,-AO  pereent  o|  .software 
"fforts  are  devoted  to  . hei  kout  and  testing,  and  that  oulv 
ills  nit  JO  (lereent  of  the  time  is  spent  ill  coding  I'lie  data 
base  for  these  figures  was  derived  from  larg"  s \ stems  (irojee’s. 
such  as  t lie  t >S  dliO  development  Ours  arc  nturli  more  inodes' 
projects,  and  our  results  an  l oth  different  and  more  variable 
About  .VO  | lereent  of  our  time  lit  the  s\ nth'  t ie  library  (irojt i t 
was  devoted  to  coding,  and  less  than  Jo  pi  rn  tit  of  the  lime 
was  spent  on  integration  and  testing.  I'lie  Benehmark 
I’reparatioii  System  hgures  are  ipnte  different  Coding  has 
taken  up  leas  than  Jo  percent  of  our  time,  with  integration 
and  testing  using  up  sonic  HO  percent  of  tie-  time  I'lie  result- 
ing low  ligure  for  analysis  and  design  ! percent)  is  due  to 
the  simple  fact  that  much  is  already  known  about  the  port- 
ability problem  areas  in  COBOL 

Packaging  and  distribution  of  all  our  software  product-  fol- 
low fairly  simple  guidelines  The  programs  in  machine  net 
pendent  form  tall  implementor  names  are  parameteriz«xl,  as 
are  machine  dependent  features  such  as  precision  and  size  of 
numerical  fields),  are  placed  on  a standard  magnetic  tape 
risd,  together  with  a copy  of  the  VP-Routine  The  latter  is 
usisl  for  parameter  substitution  (to  a form  acceptable  to  any 
specific  system),  "library  management",  ant)  job  control 
statements  generation  Accompany  mg  tin  tape  is  a user 
guide,  brief  narrative  description  of  the  system,  and,  where 
applicable  experimental  results.  The  programs  arc  self 
documented,  so  that  wc  can  avoid  excessive  external  docu- 
mentation While  wc  recognize  the  ini|strtance  of  ailet/unJe 
documentation,  wc  have  found  that  excessive  documentation, 
such  as  detail. si  How  charts  can  Is-  a hindrance  to  proper 
documentation  Distribution  is  through  the  .National  Tech- 
nical Information  Service 

A few  words  of  caution  aliout  these  and  other  published 
statistics  ami  practices,  h irst  of  all,  they  reflect  a very  sjs  cilic 
environment.  We  have  a small  (eight  people)  staff  with  veri 
homogeneous  backgrounds  Our  systems  are  modest  tit  size, 
and  "utility”  orientisl  All  our  work  muni  lie  |sirtahle,  since 
we  are  currently  using  l’NIVA(*-l  I Off.  IBM  Httf)  (in,  and 
I1IS  tlO.V)  systems  for  product  development  Furthermore, 


. ur  t i >Ht  >1 , compiler  vaboation  re-  j>oiisibilith*8  bav<*  re- 
in: !v  r « |iiir» •. ! ' •.»  u**  our  sm'twan  on  a Burroughs  6700, 

IIIS  4-<7  and  IBM  bO  ] V»  system  Thus,  portability  is  truly 

a neceasit  \ for  its 

Secondly  * \ • f !•-’•  a -ni  il.tr  cnv iroium  nt , the  statistics 
h uNl  1"  w i as  'guidelines  They  am  simply  products 
of  i ur  experience  which  we  hope  to  learn  from  but  do  not 
export  to  be  bound  by 

Till.  I’AVOl  1 
What  has  it  all  cost  us? 

Total  cos;  f..r  tb  synthetic  programs  library,  including 
n .r  bin*  tim<  clerical  suppor*  and  salaries  was  under 
$n.i Nit)  This  benchmark  preparation  conversion  package  has 
<«  -■  r-  about  SS.OflO  lie  (’ORt >1.  Compiler  Validation  Sys- 
tem \u>  originated  in  P.Miti  by  the  I S.  Navy  Programming 
l^ttiguag' > Croup  under  the  direction  of  (’apt  Grace  M 
Ibipp*  r.  I'SNB  \ reasonable  estimate  of  its  initial  cost  is 
leit  possible,  but  At  <|. . have  an  expecteil  cost  for  the  audit 
routines  \x < are  preparing  in  antieipation  of  the  revised 
t y >U»  >1  -’an  lard.  t>ur  schedule  rails  f*>r  completion  of  the 
project  by  NoveiubcT,  BI74  dotal  calendar  time  for  the 
project  will  be  1 * months  and  w«*  anticipate  to  expend  3b 
man-months  • ti  the  effort  I ota!  » .-t  for  the  new  Validation 
System  should  be  in  tie  neighbor!.  .*»*!  of  S7.VOOO  The  new 
system  vsill  be  approximately  twice  the  size  of  the  current 
one,  which  is  comprised  o*  about  l ‘i(|  programs.  or  100.000 
lines  t*f  ( ’( )B<  d.  ci ale  Th*  implication  lier*  is  that  we  ex|*ect 
our  productivity  to  be  about  M.OOO  lines  of  COBOL  code 
per  man-year  a remarkably  high  figure  (Corbato*  has  re- 
ported a number  in  the  in  ighlmrli.iod  of  1200  PL  1 lines  of 
rode  |ter  man-year  on  tin  Multics  project  > This  is  due  almost 
entirelv  to  the  fact  that  we  are  borrowing  most  of  the  de- 
sign work  from  the  present  Validation  System  W*  know  the 
modules  we  w ill  rerpiire  since  the  standard  is  defined  for  ur 
The  VP- Routine  is  already  available  Many  of  the  audit 
routines  will  l»e  < xt melons  of  current  ones  Thus,  our  time 
will  be  spe'it  primarily  in  identifying  tests,  c< sling  and 
testing 

What  are  the  benefits? 

We  expect  the  returns  «*ii  our  investment  to  be  substantial 
The  best  COBOL  compiler  we  have  tested  to  date  ('“liest" 
in  its  conformance  to  the  standard  has  had  some  HO  areas  nf 
iioii-conforiuanee  I bis  not  only  impacts  jmrtability,  but  can 
have  serious  side  efforts  Many  data  base  management  sys- 
tems are  COIiOI/'baiwd  Krrors  in  a compiler  ran  easily  re- 
sult in  "dirty  data  getting  into  the  data  base  We  have,  for 
' *y.Hii i pi*  identified  some  four  different  treatments  «f  anth- 
ffictic  statements,  each  producing  different  results!  The  vali 
da t ion  of  a compiler  tells  us  where  the  danger  areas  lie 
Furthermore,  vendors  nr**  required  to  enmet  discrepancies 
once  these  have  Ihs-ii  identified  Thus,  our  validation  of 
C’(  >B<  )\.  compilers  enables  us  to  reap  the  benefits  »tf  standard 


ization  Without  mu  )i  a r.«a  uremeut  tool  ^tandardiza'  :<in 
a fruit!  ess  »•  nd  cay  ■ *r. 

The  high  cost * «<f  pr./resMisg  In  nchuiarks  has  already  b**«  i 
mentioned  Y\ <•  know  of  a remit  selection  where  the  total 
award  wa  for  approximately  $.  million  Included  r tb 
figure  w.*r«’  Mine  $.'*00,1)00  which  tin  vend  >i  pent  in  pro* 
ing  the  benchmark.  Both  tht  benchni.ok  preparation  >\  • n 
and  *»tir  synthetic  program**  library  w uld  pay  I r itself  •: 
even  a small  portion  of  tin-  potential  vtving-  in  v*nd 
expenditures  ar**  passed  bark  to  th*  Navy 

Finruis  ki  loin's 

The  benefits  d riv  d fr  m yvtlui.it itig  ( OBi'b  < inj  • 
would  also  a*  i ru*  in  the  validation  of  < : pihr-  ?■  r »th.  r 
languages  FORTRAN  BASIC  and.  lat*  r.  1*1  •:*  .a'  * 

candidates 

( ’ompiler  • fh<  i -n*  y in  terms  *.f  object  -d  • v ■■■ut  i.  ■ ; > -j .« •.  • 1 
and  storage  utilization  has  a Mgnifu  ant  impact  . an 
lation's  throughput  this  in  tun  ff«cts  1 1 timing  -v  1 
tions  and  therefore  of  expenditures  \\*  brine  more  ' 
ficicnt  compilers mean  fewer  I I In  r *pcnt , or  nue  \oikd 
fur  the  same  \loilar  A bu-.  w>  .n*  phu  mug’u  »’  *1  1 i 
tinea  to  determine  flu-  retail et  w ith  of  .»  given  <•  mp  '■ 
That  is,  w want  measure  ich 

improvement  i».  • x*-*  u’  at  sp.  **d  ..nl  >t*»rag  ? • *« i u:r* uJ  1*1  • 
projix-t  is  III  till  design  phas*  ttid  will  n •>’  n * It-'lt  ; 1 1 1 1 1 ; « : \ . 
to  FORTRAN  compiler**,  principally  be<  ei-*  t rasn  r t . . 

rneasun*  eflicienev  of  I OUTRAN  • in  pi  b i than  tli--  t«.r 

most  othet  languages  Knuth’.*  work1*  sugg.  ***s  that  • "i  ef 
forts  may  prove  fruitful 

Finally,  w*  believ*  that  seri*»us  th  .ugtit  must  b*  giy.  » t • 
validating  gen*  rallies!  data  base  n.  iiiftgrm.  fit  s>sten 
Specifically  we  ar.  inter«aste*i  in  finding  y . - •*-  rt  aenr 

that  th  data  base  n build.**  with  th*  • . d«*  *•  m*i  *sl 

contain  what  wr  wish  i;  to.  tl.  it  in  r- trn  vmg  *I.P  » • 1 . 

that  is  profMT,  and  •**..’/  what  i-  prop*  i .*n  i tf  a u ■ f i, 
Hyst«*ins  does  not  impart  tb*  mt  g w\  ••  *l»*  *b»la  l*a>  \N 
also  plan  t«.  *l**v«  i«  p smipl**  *n  * i \ « • al  n d !-  t I..  • d . 

evaluating  diff**r* ut  typ***- of  dai. * igamzati  a**  I - • *1- 

nng  ing  coT  tanunat i- n *«f  thes*  *lat«  l a--  •*  n i i' 

o\>je»u  ciai**  has  air* -ai|y  been  * nmu  niid  • 

GON(  ’l  l SB  »N 

Software  t<»  be  us«s|  in  improving  *f  : -•  asuring  tb**  *pe*lity  t 
other  software  is  neither  difficult  n**r  expensive  to  produc* 
Our  efforts  ar*‘  cimcent rated  in  th*  svst**m  selection  area. 
We  b«*lieve,  howeviT,  that  flu  InTetita  t«*  t*<*  d«*rivtd  from 
such  efforts  have  a broader  scope  and  are  substantial  . imugh 
to  warrant  js*r?*u:d  by  any  data  *roc«  s*tng  organization 
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t.  Ahs'r.u  is 


Three  software  tools  developed  by  the  Federal  COBOL  Compiler  Testing  Service  for  use 
in  system  quality  control  are  described:  a COBOL  Compiler  Validation  Svstem  being 

used  throughout  the  Federal  Government  to  determine  the  degree  to  which  COBOL 
compilers  conform  to  the  COBOL  Standard;  a system  to  facilitate  the  process  of  COBOL 
programs  conversion;  and  a set  of  synthetic  programs  for  system  performance 
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